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ABSTRACT 



Software project development has been plagued with an infamous 
reputation for cost overruns, late deliveries, poor reliability and 
users' dissatisfaction. Much of this blame has been placed on the 
manner in which software development projects are managed. The 
System Dynamics Model of Software Project Management is a 
quantitative model of software project dynamics that is attempting 
to gain some valuable insight into the managerial side of 
developing software systems. 

The objective of this thesis is to use the System Dynamics 
Model's gaming interface to investigate the effects of feedback on 
software project managers. Specifically, subjects were provided 
with either feedforward, outcome feedback, or cognitive feedback to 
determine which feedback form, if any, improved the subjects' 
performance when confronted with a complex dynamic task, such as 
software project management. The results show that subjects in the 
cognitive feedback condition achieve a higher level of performance 
than those in either the feedforward or outcome feedback 
conditions . 
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INTRODUCTION 



I . 



A . BACKGROUND 

The proliferation of computing equipment over the past 
years has served to increase the demand for more reliable and 
complex software. Unfortunately, the success that has been 
common to the hardware industry has not been shared by those 
in the software industry. Today's software projects are 
typically delivered late and over budget. These inaccuracies 
have been blamed, in part, on ineffective software project 
managers. (Schlender, 1989) 

A large portion of these inaccuracies associated with the 
general project management problem can be attributed to the 
difficulty of control. One basic element, evident in any 
control system, is a means of transmitting feedback 
information to the control device (Anthony and Dearden, 1980, 
pp. 3-4). Control relies heavily on information feedback; the 
question is, however, what kind of feedback? 

There has been a great deal of research analyzing the 
effects of outcome feedback on management control, but this 
type of feedback has not been effective in improving the 
performance of decision makers. Research in static situations 
shows cognitive feedback to be more effective than outcome 
feedback in enhancing decision quality. However, little work 
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has been done to determine how cognitive feedback may assist 
management control in complex dynamic tasks. 

Researchers have also suggested that the performance of a 
decision maker in a dynamic decision task, such as software 
project management, would improve if the decision maker's 
model matches that of the task. Therefore, providing 
individuals with feedforward on a task may improve decision 
quality as opposed to outcome feedback. The focus of this 
thesis is on studying the effects of outcome feedback, 
cognitive feedback, and feedforward on one particular 
management control problem, that of software project 
management . 



B. EXPERIMENTATION TOOL 

The System Dynamics Model of Software Project Management 
(SDM) is a comprehensive model of the software development 
process that integrates both the managerial and software 
development activities. Through the use of a model. 

The effects of different assumptions and environmental 
factors can be tested. In the model system, unlike the 
real systems, the effect of changing one factor can be 
observed while all other factors are held unchanged. Such 
experimentation will yield new insights into the 
characteristics of the system that the model represents. 

By using a model of a complex system, more can be learned 
about internal interactions than would ever be possible 
through manipulation of the real system. Internally, the 
model provides complete control of the system's 
organizational structure, its policies, and its 
sensitivities to various events. (Forrester, 1961, p.l) 
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Additionally, this particular model provides an effective 
means of studying dynamic decisions. 

The gaming interface of the System Dynamics Model provides 
experimenters with the ability to analyze the efforts of any 
number of software project managers. The experimenter can 
vary the type of feedback given to the manager by specifically 
tailoring the model's interface for that particular manager. 
The model provides the capability of displaying a wide variety 
of variables in either tabular or graphical form. The results 
of each manager's run can then be collected and analyzed to 
determine any particular trends in their decision making 
process . 

C. RESEARCH QUESTION 

Recent laboratory experiments have provided valuable 

insight into human behavior in a variety of decision- theoretic 

contexts. This research, however, has focused mainly on 

static and discrete judgements. As Hogarth (1981) emphasizes 

...the continuous, adaptive nature of the judgmental 
processes used to cope with a complex, changing 
environment.... With few exceptions ... judgment 

researchers have focused on discrete incidents (particular 
actions, predictions, and choices) that punctuate these 
continuous processes; furthermore, task environments are 
typically conceptualized to be stable, (p. 198) 

Sterman (1989a) argues that experimental studies of the 
"continuous, adaptive nature of judgmental processes" in a 
dynamic system, such as software project management, can be 
conducted in the laboratory with the aid of computer 
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simulation models. He adds that "simulations can represent 
the structure and complexity of such systems with great 
fidelity and permit controlled manipulations of the decision 
context and information presented to the subject." 

As an example, the research question addressed in this 
thesis is: What effects do cognitive feedback, outcome 
feedback, and feedforward have on decision makers in a dynamic 
decision environment such as software project management? 

D. CONTRIBUTION 

Enhancement of management control through the use of 
cognitive feedback has attracted much attention (Kleinmuntz, 
1985; Sterman, 1989a). The use of cognitive feedback to aid 
software project managers has not, however, been investigated. 
The goal of this research, therefore, is to establish the 
importance of cognitive feedback as an aid to the decision 
making process of software project managers. 
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THEORETICAL PREMISE 



II . 

A. FEEDBACK IN A DYNAMIC DECISION ENVIRONMENT 

1. Static vs Dynamic Decision Environments 

When analyzing human judgmental ability, it is 

important to distinguish between static and dynamic decision 

environments. Although much of human decision making is 

composed of discrete incidents (particular actions, 

predictions, and choices) occurring in a seemingly static 

environment, these incidents are only a subset of, and serve 

to punctuate, our continuous processes which occur in response 

to our dynamic environment. As Hogarth (1981) indicates, the 

limitation of existing research on human judgment is that it 

focuses only on these discrete incidents in static 

environments. Since most decisions are made in a continuous, 

dynamic environment, it is argued that biases observed during 

these discrete incidents occur as a result of heuristics that 

are derived from man's more natural continuous environment. 

According to Hogarth (1981), failure to evaluate human 

judgement as a continuous process has two distinct pitfalls: 

First, insufficient attention has been paid to the effects 
of feedback between organism and environment. Second, 
although judgmental performance has been evaluated 
according to the principles of optimal behavior implied by 
decision theory and the probability calculus, few 
researchers have questioned whether the assumptions of 
such models apply to continuous processes, (p. 198) 
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Studies involving human decision making cannot 
overlook the importance of feedback. Most all human judgement 
is used to facilitate an action and this action is most often 
followed by immediate feedback. Our next action is then 
directly influenced by this feedback causing a action, 
outcome, feedback, action loop. Hogarth (1981) indicates that 
the tendency to overlook feedback as a crucial part of this 
loop comes as a result of its "ubiquity" in the environment. 
As Powers states (1973): 

All behavior involves strong feedback effects, whether one 
is considering spinal reflexes or self-actualization. 
Feedback is such an all-pervasive and fundamental aspect 
of behavior that it is as invisible as the air we breathe. 
Quite literally it is behavior--we know nothing of our own 
behavior but the feedback effects of our own outputs, (p. 
351) 

2. Inadequacy of Outcome Feedback 

A majority of the work that has been done in relation 
to the dynamic decision environment has examined the effects 
of outcome feedback on the decision making process (Brehmer, 
1987; Sterman, 1989b). Evidence, however, indicates that 
presenting outcome feedback in a dynamic environment has 
dysfunctional effects that persist over time. These effects 
fall into four categories. 

First, subjects typically misperceive time lags in the 
system which confronts them. In Sterman 's (1989b) stock 
management problem, subjects fail to adequately account for 
the supply line. Subjects confronted with Brehmer 's (1987) 
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DESSY experiment show improvement after spending two hours a 
day for four days with the system, "but only if there are no 
delays in the system. If there are even minimal delays, the 
subjects' control over the system does not improve." As 
Brehmer states, "this [fact] is somewhat disconcerting, since 
delays are probably a more common case than that of immediate 
feedback . " 

The second dysfunctional effect that typically plagues 
subjects in outcome feedback experiments is a wide oscillation 
of results over time (Sterman, 1989b). In Sterman's stock 
management problem, this oscillation is seen in the inventory 
level. Subjects in Sterman's experiment also attribute the 
dynamics of the system to external variables rather than as a 
direct result of their interactions with the environment. 
Thus, subjects misperceive the feedback from their own 
decisions. The final dysfunctional effect of outcome feedback 
is seen in (Wagenaar, 1985) where subjects misperceived 
exponential growth over time (and hence, nonlinear changes). 

Experimental evidence indicates that outcome feedback 
is not an adequate aid in decision making. As Sterman (1989b) 
states: "The results here suggest that outcome feedback alone 
is not sufficient: by attributing the source of change to 
external factors, people's mental models lead them away from 
the true source of difficulty." Kleinmuntz and Thomas (1987) 
drew similar conclusions: "Despite the corrective benefits of 
outcome feedback. ... it may still be quite difficult to learn 
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how to improve one s decision rules using outcome feedback 
alone." The inadequacy of outcome feedback in dynamic 
environments has led researchers to explore alternative means 
of improving performance. 



B. ALTERNATIVES TO OUTCOME FEEDBACK 

1 . Cognitive Feedback 

In contrast to outcome feedback, which provide 
subjects with information about the accuracy or correctness of 
their response, cognitive feedback represents "information 
regarding the how or why that underlies this accuracy." 
(Jacoby et al., 1984) Doherty and Balzer (1988) describe 
cognitive feedback as information which provides subjects with 
the following relationships: 



1. Between cues and criterion, i.e., information about the 

task. This is known as task information and it is 

characterized by three kinds of relational indices-- 
overall task predictability (Re), cue intercorrelations 
(rij), and correlations between cues and the criterion 
(rie). 

2. Between cues and the person's inferences, i.e. 
information about the person's cognitive state. This is 
known as cognitive information and, in terms of the lens 
model, largely mirrors task information (the only 
exception being cue intercorrelations which is not 
represented in this relationship). 

3. Between cognitions and the distal objects. This third 
category comprises indices of "functional validity" 
information, or, information about the relation of the 
cognitive system to the task system (Balzer et al., 
1989). These indices include the achievement correlation 
(ra), the matching index (G), and the correlation between 
the residuals from the predictions of those models (C). 
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Since Cognitive Feedback has been used largely for 
static tasks in the context of the lens model, the aim of this 
study is to extend this notion to a dynamic decision situation 
using a system dynamics model. As Sterman's (1989b) research 
indicates "the efficacy and robustness of decision strategies 
lies not only in the availability of outcome feedback, but 
depends crucially on the nature of the action feedback between 
decisions and changes in the environment which condition 
future decisions." Brehmer (1987) concludes from his DESSY 
experiment that, "results on verbalization suggest that 
information about the system may need to be communicated in 
nonverbal form, and that various graphic displays may prove 
useful." Additionally, Kleinmuntz and Thomas (1987) state 
that "feedback about the decision process being used may also 
be valuable...." Each of these statements support the belief 
that cognitive feedback should prove more beneficial than 
outcome feedback when presented to subjects confronted with a 
dynamic decision situation. 

2. Feedforward 

Another method of assisting subjects in forming mental 
models of complex systems is through feedforward. Feedforward 
can be defined as "the transmission of task information 
directly to the subject." (Bjorkman, 1972) Studies by Conant 
and Ashby (1970) showed that in order to perform effectively 
with a dynamic process, an operator must have a model of the 
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system. Bjorkman (1972) provides three hypotheses with regard 
to the use of feedforward in knowledge and policy formation: 



1. Knowledge acquired by feedforward should be more accurate 
and consistent since it does not suffer from various 
sources of error and bias due to the trial--by--trial 
accumulation of information. 

2. Feedforward relieves the learner from a certain amount of 
cognitive strain since he already knows things which 
otherwise should have been learned by feedback. This may 
give the learner an increased opportunity to focus on 
policy formation. 

3. It seems reasonable to assume that feedfor rd favors an 
analytical rather than intuitive mode of thought. 
Uncertainty, which is one of the factors that contributes 
to intuitive rather than logical, stepwise inference, has 
been removed entirely or partly by feedforward. 



The operationalization of feedforward occurred, for the 
purposes of this study, through the use of a special training 
session which assisted subjects in forming a model of the 
software project management system. 

3 . Cognitive Feedback vs Feedforward 

Although feedforward assists subjects in forming a 
mental model of the system, task information is presented only 
prior to performing the task (or, at best, the information is 
presented prior to performing the task and the same 
information is available to the subject throughout the task). 
As shown by Morris and Rouse (1986), "a priori knowledge can 
be a powerful basis for gaining new knowledge or, if 
incorrect, an impediment to gaining correct knowledge.” A 
distinct advantage gained through cognitive feedback is that 
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subjects are constantly being presented with task information 
so that they may change their mental models of the system to 
meet changes experienced in the environment. 

C. HYPOTHESES 

Two primary hypotheses guide the research question. The 
first hypothesis is concerned with the relationship between 
feedback condition and subject performance. As indicated in 
sections 1 and 2, cognitive feedback and feedforward assist 
subjects in forming a mental model of the task. Thus, 
subjects receiving this information should perform better than 
those not receiving this information. Additionally, cognitive 
feedback continually assists subjects in keeping their mental 
model current with the dynamic environment. One would 
therefore expect subjects receiving cognitive feedback to also 
outperform those subjects receiving feedforward. This leads 
us to the first hypothesis: 

Subjects receiving cognitive feedback will perform better 
than subjects receiving either outcome feedback or 
feedforward . 

The second primary hypothesis addresses the characteristic 
of the task which confronts the subject. This is accomplished 
by measuring each subjects performance during each of three 
software projects with varying degrees of complexity (details 
on the three projects, referred to as ideal, f ixedsize/bad 
estimates, and undersize will be provided in Chapter III). We 
would expect that subjects would perform better when 
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confronted with a task with less complexity. Since the ideal 

project has fewer lags and less of the "noise" associated with 

more complex tasks, it is regarded as the least complex of the 

three projects. The second hypothesis is therefore: 

Subjects performance while managing the ideal software 
project will be better than when managing either the 
f ixedsize/bad estimates or undersize software projects. 
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Ill . 



METHOD 



A. TASK ENVIRONMENT 

The task that subjects were asked to perform was in many 
ways similar to flight simulators that pilots use to mimic 
flying an aircraft from takeoff at point A to landing at point 

B. Instead of flying an aircraft, though, the simulation 
mimicked the life of three real software projects from the 
start of the design phase until the end of testing. Subjects 
were more than outside observers, however, they performed an 
actual role in the project: that of the project manager. 

Specifically, subjects were required to track each 
project's progress using a number of reports generated by the 
project team at different intervals throughout the project 
life. They then made project staffing decisions based on the 
knowledge gained from those reports. As project manager, 
subjects were permitted to hire additional staff or decrease 
the staffing level as deemed necessary to complete the 
project. Their objective in setting the staffing level was to 
decide on the best compromise between finishing on an 
acceptable schedule while avoiding an excessive cost overrun. 
Specifically, subjects attempted to: 

1. complete the project on schedule, 

2. at the lowest possible cost, and 
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3. in any case, complete it before the maximum tolerable 
completion date. 

Unlike Sterman's (1989b) use of a Generic Stock-Management 
System and Brehmer's (1987) DESSY experiments which presented 
subjects with open-ended tasks, the task which confronted each 
subject in our experiment was close-ended with each project 
having a finite completion point. 

The task of managing a Software Project was selected for 
several reasons. First, Software Project Management has all 
of the characteristics of a dynamic problem (Brehmer and 
Allard, 1985): 



1. It requires a series of decisions. 

2. The environment changes both spontaneously (staff 
productivity, changing requirements, etc) and, as a 
consequence of the decision maker's actions. 

3. The time element is critical; it is not enough to make 
the correct decisions and to make them in the correct 
order, they also have to be made at the correct moment in 
time . 



Like Brehmer's (1987) DESSY experiment, the Software Project 

Management problem is interesting 

. . .because the standard normative theories for decision 
making do not apply (Brehmer and Allard, 1985); the models 
of the task embodied in these theories simply do not fit 
this kind of task. It is not possible to compute the 
correct course of action. This can only be found from a 
model of the system and, before the operators have 
developed such models, they will not be able to control 
the system. The research problem, is whether or not 
people are able to develop good mental models of this and 
similar tasks, (p. 24) 
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Finally, Software Project Management is currently a critical 
issue due to the frequency of projects that are delivered over 
budget and late (Schlender, 1989). 

Three separate projects, referred to as ideal, 
f ixedsize/bad estimates, and undersize, were selected in an 
attempt to cover the spectrum of projects that typically 
confront Software Project Managers. Table 1 shows the 
characteristics associated with each of the three projects. 



TABLE 1. PROJECT CHARACTERISTICS 





Ideal 


Undersize 


Fixedsize 


Project costi initial estimate 
( Man-days ) 


3,721 


1 ,460 


2,972 


Project Size* initial estimate 
(No. of tasks) 


1 ,067 


397 


1 ,866 


Actual Size of Project 
(No. of tasks) 


1,067 


610 


1 ,866 


Project duration! initial estimate 
( Days ) 


413 


362 


380 


Maximum tolerable project duration 
(Days ) 


479 


420 


441 



Notes i 1. The ideal project had accurate initial estimates of project size and cost. 

2. The undersize project had understated initial estimates of size, and therefore, 
cost . 

3. The fixedsize project had an accurate initial estimate of size. The initial cost 
estimate was, however, understated. 



Subjects were presented with accurate initial estimates as 
well as accurate information throughout the entire lifecycle 
of the ideal project. Subjects were given accurate initial 
estimates for the fixedsize/bad estimate project, however, 
estimates given during the project lifecycle were typically 
unreliable. The undersize project was a project that grew in 
size from an initial estimate of 397 tasks to 610 tasks at 
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project completion. This growth in project size was 
attributable to changing user requirements. 

B . MODEL 

The Model of Software Project Management attempts to 
provide "a comprehensive model of the dynamics of software 
development that enhances our understanding of, provides 
insight into, and makes predictions about the process by which 
software development is managed." (Abdel-Hamid and Madnick, 
1989) Figure 1 shows the model with its four subsystems: the 
human resource management subsystem, the software production 
subsystem, the controlling subsystem, and the planning 
subsystem. "The model was developed on the basis of a battery 
of 27 field interviews of software project managers in five 
software producing organizations, supplemented by an extensive 
database of empirical findings from literature." (Abdel-Hamid, 
1989) The human resource management subsystem accounts for 
variables related to the workforce, namely, the hiring rate, 
training, and turnover of project personnel. The software 
production subsystem models the designing, coding, and testing 
phases of the software development lifecycle. This subsystem 
also accounts for the quality assurance effort required for 
project develop as well as the actual productivity of the 
project team. In contrast to actual productivity, perceived 
productivity is described in the control subsystem. Perceived 
productivity directly influences a manager's estimate of 
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Figure 1 . Model Structure 
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project tasks perceived to be completed. This estimate, 
however, is often unrealistic with regard to software 
development since one must have accurate knowledge of rates of 
accomplishment and resources expended to date (Abdel-Hamid and 
Madnick, 1989). Thus, this variable often is no more than a 
measurement of budgeted resources that have been expended. 
The planning subsystem, the final subsystem of the model, 
provides initial project estimates such as project cost, 
schedule, and staffing. As the project continues through its 
lifecycle, these estimates will change to reflect management's 
decisions . 

C. EXPERIMENTAL DESIGN 

The research design is illustrated in Table 2. The 
experiment used a factorial design with two components in 
order to capture the feedback condition and the project type. 
These components are between- subjects and within- sub j ects . 

TABLE 2. EXPERIMENTAL DESIGN 



Type of information 

Cognitive Feedback Outcome Feedback 

Project Ho. Project Ho. 

12 3 12 3 

Order 



Order 1 IUF IUF IUF 

Order 2 UIF UIF UIF 

Order 3 FUI FUI FUI 



Hotesi 1. Participants were randomly assigned to one of the feedback conditions and one of 
the sequences of task conditions. 

2. I * U, and F refer to ideal* undersize* and fixedsize projects* respectively. 



Feedforward 
Project Ho. 

1 2 3 
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1. Between-Sub jects 

The fundamental objective driving the experiment is to 
determine the effect of cognitive feedback, i.e., determine 
how best the operator can be conveyed a model of the system 
over time (Brehmer, 1987): through feedforward, outcome 

feedback, or cognitive feedback. This is the rationale for 
the between- subject component. 

2. Within-Subjects 

In addition to determining if systematic differences 
exist among experimental conditions, feedback studies also 
seek to study the effect within each condition over time. 
This is referred to as wi thin-subject design (Barlow and 
Hersen, 1984, p. 66) and involves multiple measurements over 
different points in time. In this experiment, the within 
subjects aspect was operationalized by using three separate 
projects, namely, the ideal, f ixedsize/bad estimate, and 
undersize projects. 

Randomization between and within subjects was achieved 
using a Latin Square Design as follows (Kirk, 1982, pp. 311- 
312) : 

First, each project was assigned a corresponding letter. 

A: Ideal Project(IL) 

B: Fixedsize/bad estimate Project(FB) 

C: Undersize Project(UN) 

Next, two sequences of random numbers were generated. 

(2,1,3) (3,1,2) 
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The appropriate square is then selected. 

ABC 
B C A 

CAB 

Rows are then ordered according to the first set of random 
numbers . 

B C A 

ABC 
CAB 

Next, columns are ordered according to the second set of 

random numbers. 

ABC 
CAB 
B C A 

Finally, the notation is converted according to project name. 



G1 


G2 


G3 


IL 


FB 


UN 



Task 


UN 


IL 


FB 


Order 


FB 


UN 


IL 



Therefore, Group 1 receives the projects in the order: ideal, 
undersize, and f ixedsize/bad estimate. 

D . SUBJECTS 

The experiment was conducted using 56 graduate students. 
Participants were divided into nine groups based on the 
feedback condition and task order. Table 3 shows the 
feedback condition and task order provided to each group. 

Each subject was assigned a number from 1 to 56 according 
to the alphabetical order of his last name. Two digit random 
numbers were then generated using a random number table. 



20 



TABLE 3. GROUP ASSIGNMENTS 



GROUP NUMBER 



FEEDBACK CONDITI ON/TASK ORDER 



1-1 

1-2 

1- 3 

2 - 1 
2-2 

2- 3 

3- 1 
3-2 
3-3 



Cognitive Feedback, G1 Task Order 

Cognitive Feedback, G2 Task Order 

Cognitive Feedback, G3 Task Order 

Outcome Feedback, G1 Task Order 
Outcome Feedback, G2 Task Order 
Outcome Feedback, G3 Task Order 
Feedforward, G1 Task Order 
Feedforward, G2 Task Order 
Feedforward, G3 Task Order 



Subjects corresponding to the first six numbers were assigned 
to group 1-1, the next six in group 1-2, etc. Duplicates and 
random numbers greater than 56 were disregarded. Due to the 
number of subjects not being evenly divisible by nine, groups 
1-3 and 2-3 had seven subjects each. 

1. Participant Profiles 

Two types of subject characteristics could potentially 
have affected results in this experiment: demographic factors 
and task-specific factors. Demographic factors were 
operationalized as age, full time work experience, years since 
completion of undergraduate education, familiarity in working 
with computers and hours per week a subject spent working on 
computers. Table 4 profiles the subjects with respect to 
demographic factors. The task-specific factor was 
operationalized by asking subjects whether they had any prior 
experience in the task. It was determined that none of the 
subjects had any significant experience in software project 
management . 
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TABLE 4. PARTICIPANT PROFILES. (Means). 



By feedback 


condition « 








Cognitive 


Outcome 






Feedback 


Feedback 


Feedforward 


AGE 


34 


33 


31 


WK EXP 


14 


13 


10 


Y UGRAD 


10 


10 


8 


FAM COMP 


5 


6 


5 


HRS COMP 


10 


9 


9 



By project order i 





Order 1 


Order 2 


Order 3 


AGE 


34 


30 


34 


WK EXP 


14 


10 


13 


Y UGRAD 


10 


8 


11 


FAM COMP 


6 


5 


5 


HRS COMP 


12 


8 


9 



Key* AGE = Age of subjects (Years) 

WK_EXP = Full time work experience (Years) 

Y_UGRAD = Years since completion of undergraduate education 

FAM_COMP = Familiarity of subjects with computers (1 = not familiar > 9 = very familiar) 
HRS_COMP = Hours per week spent using computers 



E. INFORMATION PROVIDED TO SUBJECTS 

Subjects were provided different types of information 
based upon the feedback condition corresponding to their 
group. During the lifecycle of each project, the experiment 
software would pause at 40 day intervals to allow subjects to 
review this information. 

1 . Outcome Feedback 

Subjects receiving outcome feedback were given only 
one report, the Project Status Report, at the end of each 
interval. Information provided in this report is shown in 
Table 5. 
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TABLE 5. PROJECT STATUS REPORT 



CURRENT INTERVAL STATISTICS. Elapsed Time = 

INITIAL ESTIMATES. (These will not change throughout 


80 

the project ) 


Days 


Project size 


1 ,067 


Tasks 


Project duration 


415 


Days 


REPORTED STATISTICS at Time =====> 


80 


Days 


V. Development Reported to be complete 


10.26 


Percent 


Y, Testing Reported to be complete 


0.00 


Percent 


Perceived Total Project Size at this point 


1 ,066.67 


Tasks 


Perceived Total Project Cost at this point 


3,721.00 


Man Days 


Total Humber - Fulltime Equivalent Staff 


<♦.3 


Fulltime staff 


New Estimate of Project Duration (start - end) 


829 


Days 


Maximum Tolerable Completion Date 


<♦79 


Days 


Total Man Days Expended 


348.36 


Man Days 


Total Number of Tasks developed to date 


137.45 


Tasks 



PRESS <ENTER> TO RETURN TO MENU 



2. Feedforward 

In addition to receiving the Project Status Report 
(Table 5), subjects in the feedforward groups were given a 
separate training lecture prior to the experiment which 
provided further insight into the human resource management 
subsystem of the Model of Software Project Management. Figure 
2 is an exploded view of this subsystem. 

The first part of the feedforward training provided 
subjects with instruction on two concepts critical in the 
human resource management subsystem: average productivity and 
net cumulative contribution. To demonstrate the importance of 
carefully considering each of the two concepts, the following 
human resource management problem was given: 

The initial project team consists of five people each 
with a productivity of ten lines of code (LOC) per man day 
(MD) thus giving a total output of 50 LOC per day for the 
entire team. The assumption is that the project is behind 
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Figure 2. Human Resource Management Subsystem 



schedule and the manager must make a decision either to add a 
new person to the team or accept the schedule slippage. 

Case one examines the effect of adding a new person 
with a productivity of 8 LOC per man day. If this person is 
added, it is expected that the productivity of the old team 
will decrease by 10 percent (i.e., 9 LOC/MD) due to training 
and the added communication overhead. Case two also adds a 
new person but this person's productivity is only 4 LOC per 
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man day. Again, this will cause a 10 percent decrease in the 
productivity of the old team. 

In the first case the output of the team increases to 
53 LOC/Day (given by 5 team members x 9 LOC/MD + 8 LOC/MD for 
the new member). The average productivity of the team is now 
8.8 LOC (53 divided by 6 team members). The net cumulative 
contribution of the new person is 53-50 or 3 LOC/MD. Since 
the average productivity of the team decreases, the cost of 
the project will increase, but since the net cumulative 
contribution of the new person is positive, the schedule will 
go down . 

In case two, the output of the team is only 49 LOC/Day 
(5 team members x 9 LOC/MD + 4 LOC/MD for the new team 
member). The average productivity of the team is 8.1 LOC (49 
divided by 6 team members), and the net cumulative 
contribution of the new person is 49-50 or -1 LOC/MD! Thus 
the addition of the new team member is detrimental to the 
project, not only driving up the project's cost but its 
schedule as well. 

Although the mathematics of the concepts are 
relatively simple, the importance of the lesson is to realize 
that adding a person (or people) to a late project will not 
always improve the project's schedule. The manager must look 
closely at the average productivity of the project team as 
well as the net cumulative contribution of any team members 
added . 
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The second part of the feedforward training lecture 
focused on considerations involved in the willingness to 
change workforce (WCWF). Subjects were presented with the 
equation , 

Workforce = Indicated * WCWF + Current * (1-WCWF) 
Sought Workforce Workforce 

and its relation to the WCWF curve (Figure 3). 1 The 

instructor explained to subjects that a manager at a point in 

the project which yields a WCWF value of zero from the curve 

is, in essence, not willing to change his workforce and thus, 

the workforce sought will be equal to the current workforce. 

If, however the manager is at a point in the project where the 

WCWF is one, the manager is very willing to change the 

workforce and thus, the workforce sought is equal to the 

2 

indicated workforce. 

3. Cognitive Feedback 

Subjects receiving cognitive feedback also received 
the Project Status Report (Table 5) after each 40 day 
interval. Additionally, personnel receiving cognitive 

feedback had the option to view a Cognitive Feedback Report as 
well as four plots. The Cognitive Feedback Report (Table 6) 



Subjects were told that the time parameter referred to in 
the figure was the Siam of two parameters from SDM's human resource 
management subsystem. These two parameters are the hiring delay, 
set at 30 days for this experiment, and the assimilation or 
training time, set at 20 days. 

2 

The indicated workforce is ynonymous w :h the workfcce 
necessary to stay on schedule. 
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TABLE 6. COGNITIVE FEEDBACK REPORT 



CURRENT INTERVAL STATISTICS i Elapsed Time = 


80 


Days 




INITIAL ESTIMATESi <These will not change throughout the project) 


Project Size 


1,067 


Tasks 




Project Duration 


413 


Days 




REPORTED STATISTICS at Time =======> 

Fraction of Workforce that is Experienced 


80 

0.9 


Days 




Perceived Average Productivity 
Communication Overhead 


0.4 

0.01 


Tasks/Man 


-day 


Total Number - Fulltime Equivalent Staff 


4.3 


Fulltime 


Staff 


Estimated Workforce Needed to Stay on Schedule 

PRESS <ENTER> TO RETURN TO MENU 


4.5 


Fulltime 


Staff 



provided information on specific workforce variables. Four 
plots, described as the Project Size Plot, the Staffing and 
Schedule Plot, the Workforce Mix Plot, and the Workforce 
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Productivity Plot, were provided to assist subjects in 
spotting trends developing throughout the project lifecycle. 

The plot on Project Size (Figure 4) plotted two 
variables, the perceived total project size to date (PJBSZ) 
and the perceived total project cost to date (JBSZMD), over 
the project lifecycle. This plot provided subjects 
information on whether any schedule or budget slippage was a 
result of either unexpected increases in the project's size 
(e.g., due to changes in users' requi rements ) , or that the 
effort required to complete the project was initially 




Figure 4. Project Size Plot 
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underestimated (e.g., because the project complexity was 
underestimated). If the former case were true, subjects 
would expect to see the variable PJBSZ increase over time. If 
the latter were true, then the variable JBSZMD would increase 
over time. 

The plot on Project Staffing and Schedule (Figure 5) 
plotted three variables of the project lifecycle: the total 
number of fulltime equivalent staff (FTEQWF), the new 
estimate of project duration from start to end (SCHCDT), and 




Figure 5. Project Staffing and Schedule Plot 



29 



the estimated workforce needed to stay on schedule (WFINDC) . 
This plot provided feedback on the trade-off between 
minimizing schedule over-runs versus minimizing cost over- 
runs. When a project runs into difficulties, a manager can 
choose to stick with the project's schedule (SCHCDT) by 
increasing the workforce level ( FTEQWF ) . This practice always 
increases the cost of a project. On the other hand, a manager 
might wish to minimize his/her cost over-run, by avoiding an 
increase in the workforce level, and instead, opt to increase 
the project's scheduled completion date. The indicated 
workforce level (WFINDC) is provided as an estimate of the 
workforce needed to stay on schedule. 

The plot on Workforce Mix (Figure 6) provided subjects 
with feedback on their staffing decisions. In general, 
staffing decisions have the greatest impact on productivity. 
This option plotted three variables: the total number of 

fulltime equivalent staff (FTEQWF), the fraction of the 
workforce that is experienced (FRWFEX), and the planned 
workforce (PLANWF) over the project lifecycle. This plot was 
deemed useful for two reasons. First, larger workforces will, 
in most cases, be less productive because of the increases in 
communication and training overheads. Second, the workforce 
mix (i.e., percent of experienced vs new staff in the 
workforce) will also have an impact on productivity. The 
larger the percentage of experienced people in the workforce, 
the more productive the workforce. 
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Figure 6. Workforce Mix Plot 



The plot on Workforce Productivity (Figure 7) provided 
subjects with information on the average productivity of their 
team. It displayed the relationship between the total number 
of fulltime equivalent staff (FTEQWF), the perceived average 
productivity of the workforce (ASSPRD), and the communication 
overhead associated with the workforce (COMMOH) throughout the 
project lifecycle. The basis for presenting this plot was, 
again, twofold. First, the larger the workforce, the lower 
the average productivity. This is due to the time people 
"waste" in communicating with teammates. Second, the 
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communication overhead curve shows subjects exactly what 
percentage of a person's time is wasted in communication with 
others . 




Figure 7. Workforce Productivity Plot 
F. EXPERIMENTAL SETTING 

The experiment was conducted in two computer labs with two 
attendants per lab. Each subject was assigned a specific 
terminal and was given documentation (see Appendix) and a disk 
according to his/her group assignment. Subjects were given 
time to read over the documentation and ask questions about 
the conduct of the experiment. Questions pertaining to the 
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task were not permitted. After reading the documentation, 
subjects ran several intervals of a "dummy" project to 
familiarize them with the reports, plots and keystrokes 
required to traverse through the screens they were presented 
with. Attendants assisted the subjects with any technical 
difficulties. After completing the trial run, subjects were 
permitted to proceed with each of the three projects at their 
own pace. 

G . DEPENDENT MEASURES 

Two dependent variables, deviation from initial estimates 
and staff productivity, were used to analyze the performance 
of each subject. Two numbers were used to determine the 
deviation from initial estimates. The first number was the 
difference between the subject's completion time and the 
estimated completion time. The second number was the 
difference between the subject's final cost and the estimated 
project cost. These numbers were then averaged to yield the 
deviation (overrun or underrun) from the initial project 
estimates. Project time is defined as the length, measured in 
days, required for the subject to successfully manage each 
project from start to end. Project cost measures the 
resources expended, in man days, to complete each project. 
The staff productivity is defined as tasks per man-day and was 
determined by dividing the number of tasks associated with 
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each project by the total cost of that project ien managed by 
a particular subject. 

Lower benchmarks for time and cost were determined for 
each of the three projects by running 15 simulations for each 
project using random staff sizes. Minimum and maximum staff 
sizes for each project type were determined from subjects' 
decisions. Five hundred random numbers were then generated 
for each project to fall within the minimum and maximum staff 
size. The 15 simulations were then run for each project using 
staff sizes taken sequentially from the 500 generated in that 
project's staff range. 

H. EXPERIMENTAL RESULTS 

Tables 7 through 11 summarize the results of the 
experiment. Cases in which subjects made significant errors 
were discarded, and data from 45 subjects was retained for 
analysis. Table 7 shows performance between subjects, as well 
as within subjects, with respect to deviations from the 
initial project estimates. 

Applying the approach suggested by Winer (1971, p. 697), 
the following analysis of variance (ANOVA) model suited for 
multiple Latin Squares was used to test Hypotheses: 

Y ijk(l) m = H + 04 + + y k + Xi + 6 b + 

(*x) k i + (f«) ki + (*B) k j + e ijk(1)B where: 

H is constant, 

ctj is the sequence of the projects (i = 1,2,3), 
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Bj is the order of the project (j = 1,2,3), 

is the feedback condition (k = 1,2,3, where 1 = 
cognitive feedback, 2 = feedforward, and 3 = outcome 
feedback) , 

Xj is the type of project (1 = 1,2,3, where 1 = ideal, 

2 = undersize, and 3 = f ixedsize/bad estimate), 

6 m is the experimental participant (m = 1 , . . . , 45 ) , 
e ijk(l)m the experimental error term. 



TABLE 7. DEVIATIONS FROM INITIAL ESTIMATES. Means and (Standard Deviations). 





N 


Ideal 


Undersize 


Fixedsize 


Cognitive Feedback 


15 


0.446 


11.501 


39.29 




(1.723) 


(4.43) 


(1.854) 


Feedforward 


15 


3.858 


17.684 


47.21 






(1.794) 


(2.058) 


(5.605) 


Outcome Feedback 


15 


10.014 


25.753 


57.027 






(5.710) 


(6.052 ) 


(7.041 ) 


Random Baseline 


15 


11.12 


27.98 


56.22 






(6.12) 


(7.23) 


(9.12) 



The analysis was conducted with the General Linear Models 
procedure (SAS, 1987). Table 8 contains the ANOVA results. 
The results show that the performance of subjects across the 
three different feedback conditions was significantly 
different (F=345.89, p=0.0001). The null hypothesis of no 
significant differences among feedback conditions is therefore 



rejected. 


In other words. 


the results 


indicate 


that 


the 


subjects' 


performance was 


influenced 


by the 


type 


of 



information provided to them. 
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TABLE 8. ANALYSIS 


OF VARIANCE. 


Dependent Variable! Deviation 


from Initial 


Estimates . 


Source of 
Variance 


S.S. 


Degrees of 
freedom 


F-Value 


P 


R-Square 


Model 


50769.00 


57 


1 56 . 54 


0. 001 


0.89 


Type of 


Information 


4530 . 01 


2 


345.89 


0.0001 




Type of Task 


44166.85 


2 


3372.37 


0.0001 




Sequence 


16.53 


2 


1.26 


0.2887 




Order 


3.16 


2 


0.24 


0.7862 




Participant 


2041.19 


41 


7.60 


0.0001 




Group«Task 


4.48 


4 


0.23 


0.8126 




Group*Order 


6.67 


4 


0.26 


0.9036 




Within Groups 


504.22 


77 









Additionally, Table 8 shows that the performance within- 
subjects was significantly different depending on the type of 
project confronting them (F=3372.37, p=0.0001). The null 
hypothesis of no significant differences in performance 
depending on type of project presented is therefore also 
rejected. 

The Tukey Test for Additivity indicated no presence of 
interaction effects between the sequence of projects (p>0.2), 
or the order in which projects were completed (p>0.7). Also, 
there were no interaction effects between the type of 
information provided and the type of project (p>0.8), nor the 
type of information provided and the order of project (p>0.9). 

Table 9 summarizes tests comparing results from each of 
the feedback conditions with the random baseline with respect 
to deviations from initial estimates. The mean total 
deviation for subjects in the cognitive feedback and 
feedforward conditions was lower than the random baseline in 
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TABLE 9. COMPARISON BETWEEN EXPERIMENTAL GROUPS AND BASELINE. Dependent Variable* 
Deviation from Initial Estimates. 



Comparison 


Ideal 

Total 

deviation 


Undersize 

Total 

deviation 


Fixedsize 

Total 

deviation 


Cognitive Feedback 


r>cf b 


r>cf b 


r>cf b 


Feedforward 


r >f f 


r >f f 


r >f f 


Outcome Feedback 


n.s. 


n.s. 


n.s. 



Notes* 1. cfb* cognitive feedback) f f * feedforward, n.s.t not significant, r* random 
baseline . 

2. The comparisons represent one-tailed (directional) t-tests performed on the 
means. Thus, r>cfb indicates that the mean for that variable in the random 
baseline was higher than the mean in the cfb condition, at p<0.05. n.s. 
indicates that differences in the means, if any, were not significant at 
p<0 . 05 . 



all three projects. Subjects receiving only outcome feedback 
showed no significant performance differences from the random 
baseline in any of the three projects. 

Table 10 summarizes staff productivity results. This 
information shows the actual productivity in tasks/man-day of 
the simulated staff as managed by the subject. Table 11 
contains the ANOVA results for the staff productivity data. 
Again, there was a significant difference in performance 
between subjects (F=106.88, p=0.0001) as well as within- 
subjects (F=109.59, p=0.0001). The Tukey Test for Additivity 
indicated no presence of interaction effects between the 
sequence of projects (p>0.9), nor the order of the projects 
(p>0.7). Also, no interaction effects were present between 
the type of information provided and the type of project 
(p>0.6), nor the type of information provided and the order of 
projects (p>0.4). 
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TABLE 10. STAFF PRODUCTIVITY. Means and (Standard Deviations). 





H 


Ideal 


Undersize 


Fixedsize 


Cognitive Feedback 


15 


0.393 


0.307 


0.251 






(0.051) 


(0.047) 


(0.051 ) 


Feedforward 


15 


0.317 


0.2h9 


0.182 






(0.049) 


( 0.051 ) 


(0.049) 


Outcome Feedback 


15 


0.249 


0.177 


0.112 






(0.051 ) 


(0.057) 


(0.037) 


Random Baseline 


15 


0.238 


0.180 


0.131 






(0.060) 


(0.071 ) 


(0.032) 



TABLE 11. ANALYSIS OF VARIANCE. Dependent Variablei Staff Productivity. 



Source of 




Degrees of 








Variance 


S.S. 


freedom 


F-Value 


P 


R-Square 


Model 


1.62 


57 


8.89 


0.0001 


0.86 


Type of 
Information 


0.43 


2 


106.88 


0.0001 




Type of Task 


0.44 


2 


109.59 


0.0001 




Sequence 


0.002 


2 


0.06 


0.9443 




Order 


0.006 


2 


0.09 


0.7869 




Participant 


0.71 


41 


1.37 


0.0401 




Group«Task 


0.005 


4 


0.66 


0.6234 




Group*Order 


0.007 


4 


0.98 


0.4235 




Within Groups 


0.16 


77 









38 



IV. CONCLUSIONS 



A. SUMMARY OF RESULTS 

The objective of this study was to investigate the effects 
of cognitive feedback, outcome feedback, and feedforward on 
decision makers in a dynamic decision environment such as 
software project management. Chapter II (section B.2) pointed 
out that past research has shown the dysfunctional effects of 
outcome feedback. These dysfunctional effects have led 
researchers to seek alternative means to improve decision 
quality. Chapter II (section C) discusses two alternatives to 
outcome feedback: cognitive feedback and feedforward. Both 
cognitive feedback and feedforward seek to assist the decision 
maker in formulating a "mental model" of the task which 
confronts him/her. Without this model, decision makers have 
exhibited poor performance in handling the delays and 
oscillations associated with complex dynamic systems. 
Additionally, Chapter II (section C.3) explains why one would 
expect cognitive feedback to improve decision makers' 
performance more than feedforward. The results of this study 
support these past findings. As the analysis of variance 
tests showed, there was a significant difference in subjects' 
performance depending on the type of feedback with which they 
were provided. Subjects in the cognitive feedback condition 
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experienced less deviations from the initial project estimates 
than subjects in the feedforward or outcome feedback 
condition. Additionally, staffs managed by subjects receiving 
cognitive feedback showed greater productivity than staffs 
managed by subjects in the other feedback conditions. Also, 
tests comparing feedback groups with the random baseline 
showed that subjects in the cognitive feedback and feedforward 
conditions performed better than the random baseline in each 
of the three projects, whereas there was no significant 
performance difference between subjects receiving outcome 
feedback and the random baseline. 

B. FEEDBACK AS A DECISION TOOL 

The results of this experiment provide several 

implications for the design of project control systems, 

specifically those in support of software project managers. 

As Brehmer (1987) concluded from his DESSY experiment. 

These results show that system designers cannot rely upon 
the operators to develop good mental models f complex 
systems. This implies that we need to develop means that 
help the subjects develop such models, or possibly means 
that eliminate the need for predictive models, (p. 30) 

Brehmer provides several suggestions for improving the quality 

of information systems. The most important one, with respect 

to this study, states the need to communicate information 

about the system in nonverbal form. As this study showed, 

presenting information in graphical form did, in fact, raise 

the performance level of subjects receiving that information. 
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C. LIMITING FACTORS TO GENERAL I Z ABILITY 3 

Although a majority of the subjects in this experiment had 
some managerial experience, the question remains whether they 
had enough experience to play the game. In other words, is it 
reasonable to make a comparison between their performance and 
that of real life software managers? 

Remus (1986) found, in a study to investigate the use of 
graduate students as surrogates for managers, that no 
significant differences existed between the students and 
managers in making production scheduling decisions. Although 
software project management is somewhat different from the 
task presented in Remus' experiment, it is similar enough to 
apply his findings and assume that software engineering 
graduate students are reasonable surrogates in this 
experimental investigation. 

The next limiting factor to consider is the nature of the 
particular project environment. One should take caution 
against generalizing the results presented in Chapter III to 
all types of project situations. In this experiment, each of 
the three projects was developed in a familiar in-house 
environment i.e., what is typically described as an organic- 
type project environment (Boehm, 1981, pp. 78-82). 



3 

This section is based on an unpublished paper by Abdel-Hamid, 
Sengupta, and Ronan (1990) which describes a similar experiment 
using the SDM gaming interface. 



41 



Finally, it is difficult to claim external validity for 
laboratory-type studies. A review of the gaming literature by 
Remus (1978) does, however, indicate considerable similarity 
between decision making in games and managerial decision 
making per se. Since the project games in this experiment are 
a simulation of three real life software projects, there 
seems to be no reason why there should be any exception to 
Remus' general findings. 

D . FUTURE RESEARCH 

Based on the discussion in Section C above, one particular 
path for future research using the SDM gaming interface to 
investigate managerial decision making is evident. The above 
experiment could be replicated using real software project 
managers as the subjects. Although using graduate students as 
surrogates in research studies is useful, analyzing the 
behavior of experienced project managers could provide more 
significant and ..oteworthy results. 
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APPENDIX 



EXPERIMENTAL INSTRUCTIONS PROVIDED TO SUBJECTS 
A1 Written description given to subjects 
Introduction 

The exercise you are about to undertake is similar in many 
ways to flight simulators that pilots use to mimic flying an 
aircraft from takeoff at point A to landing at point B. 
Instead of flying the aircraft, though, this simulator mimics 
the life of a real software project from the start of the 
design phase until the end of testing. In this simulation, 
you will be more than an observer. In fact you will play a 
real role on the project: that of the project manager. 

Specifically, your role will be to track the project's 
progress using a number of reports that will be produced for 
you at different intervals during the project. You will then 
make the project's staffing decisions based on the knowledge 
you gain from these reports. As the project manager, you can 
hire additional staff or decrease the staffing level as you 
deem necessary to complete the project. Your objective (like 
that of any software project manager) is to manage your 
resources wisely and efficiently while always aiming to finish 
the project within budget and on schedule (plus any safety 
factor period available.) 



Projects 

You will be given three projects to manage, all of them real 
projects conducted in a real organization. The organization 
is on the leading edge in its software engineering practices. 
For each project, you will be given a project profile 
containing the following initial information: 

Estimated Project Size(in No. of tasks) 

Estimated Schedule Duration(in No. of Work Days) 

Estimated Project Cost(in No. of Man Days) 

Maximum Tolerable Project Duration(in No. of Work Days). 
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Your Task 



Your task is to use the reports generated by the project team 
at different points in the project to determine a desired 
staffing level for the remainder of the project. Your 
objective in setting the staffing level should be to decide on 
the best compromise between finishing on an acceptable 
schedule while avoiding an excessive cost overrun. 
Specifically, you should try to: 

(a) complete the project on schedule, 

(b) at the lowest possible cost, and 

(c) in any case, complete it before the maximum tolerable 
completion date. 

Your grade for the simulation will be based on an equal 
weighting of two factors: 

(a) The percentage by which the project overshoots the 

original schedule. Thus, if the scheduled completion date 
for the project is 200 days, and your actual completion 
date is 240 days, you will be considered to have overshot 
the schedule by ( 240-200 )/200 = 20%. 

(b) The percentage by which the project overshoots the 

original cost estimate. If the original cost estimate is 
2,000 man days and the actual cost of completion is 2,500 
man days, you will be considered to have overshot the cost 
by ( 2 , 500-2 , 000 )/2 , 000 = 25%. 

The following are some important things to consider in making 
your decisions: 

1. As the software project manager, you specify the desired 
staffing level. The actual staffing isvel may, of course, 
be different due to things you cannot control such as 
turnover and lengthy hiring delays. 

2. Each project is initialized with a particular core team of 
full time equivalent personnel (FTE). This is to reflect 
that fact that most projects start out with a small core 
team of personnel. For example, project 2 may be 
initialized to an FTE of 1.5. 

3. The hiring delay for new employees can take up to 6 weeks. 
Once new people are hired, the assimilation period for a 
newly hired employee is typically one month long. This is 
the time needed to train a new employee in the mechanics 
of the project and bring him/her up to speed. A new 
employee (i.e. one that is being trained) is only half as 
productive as an experienced employee. 
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4. The personnel turnover rate is 20% per year. 

5. At different points in the project you will be given 

reported information on the status of the project. Two 
key pieces of information for this staffing task are: (1) 

The updated estimate of the total project cost in man days 
(this update can change to reflect the addition of new 
requirements and/or changes in the estimate of the team's 
overall productivity); and (2) Effort expenditures to date 
(also in man days). Subtracting the second from the first 
yields the "Remaining Effort in man days." 

6. Let us say that at some point in the project the 
"Remaining Effort" is 1000 man days, the remaining time is 
100 days and you have 7 full time equivalent employees 
working. You are, thus, in a position where you have to 
use your judgement to do one of the following: 

1. Stick with the current schedule. If so then you will 
need a staff size of 1000/100 = 10 full time 
employees . 

2. Stick with your staff size of 7. This means the 
schedule has to be pushed back. In this case the 
model will make the appropriate adjustment to the 
schedule for you. That is extend it to 1000/7 = 143 
man days. 

3. Do a bit of both. That is increase the staff size a 
bit, say to 8, which will also mean that the schedule 
will be extended (appropriately by the model) to 
1000/8 = 125 days. 



How to Play the Game 

1. First, take some time to practice and get familiar with 
the system. 

(a) Type DUMMY for running a dummy project. 

(b) Run the dummy project for 1 interval. 

(c) Go through all the options in the menu. Please be 
sure you understand all of them. 

(d) When you are done, hit <ESC>. 
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2. The real simulation starts now. You will be given three 

projects, one at a time. When you are done with one 

project, you can move on to the next, till you have 

completed all three projects. 

3. For each project: 

(a) Insert the disk you are given, and enter a command 
from the A> prompt. The command for project 1 is 
PR0JECT1, and so on. 

(b) The system will show you the size of the initial core 
team of senior designers (the full time equivalent 
number). It will then ask you for your initial 
desired staffing level. 

(c) Next it will run through the first simulation time 
period and show you the current reported statistics. 
Make your change to the full time equivalent staffing 
level on the documentation sheet provided after 
viewing the report. 

(d) Perform step (c) for as many intervals as necessary, 
till the project is complete. A project is considered 
complete when there are less than 40 days left for the 
project to be completed. That is, 

(New Estimate of Project Duration - Elapsed Time) < 40 
days . 

Thus, if the New Estimate of Duration = 426 days, and 
the Elapsed Time = 400 days, the project is considered 
complete . 

(e) There is no need to turn in the documentation sheet 
after each interval of a project. He ^ver, A LAB 
ATTENDANT MUST VERIFY YOUR FINAL RE LTS at the 
completion of each project. Call a lab attendant as 
soon as you are done with a project. 

(f) Complete the appropriate questionnaire in your 
instruction booklet. 

(g) Move on to the next project. 



Rules of the Game 

* You will be required to provide the new desired staffing 
level for the project at the beginning of every two-month 
interval (consisting of 40 work days). The simulation 
will stop to show current reported statistics and accept 
a desired staffing level after each 40 day work period. 
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Annotate your desired staffing level on the documentation 
sheet and then enter it at the simulation prompt. 

YOU MUST WORK ALONE. You are not allowed to discuss this 
exercise with anyone other than a lab attendant. Also, 
please refrain from discussing this with any member in the 
other class until they have completed the exercise. 



Please follow the guidelines strictly. The system 
prompts, along with instructions in this booklet, will 
guide you at every stage. 

If you are in doubt about what to do next, ask for a lab 
attendant. 



Al.l Further instructions provided to cognitive feedback 

subjects 

How to use and Interpret the Plots 

Throughout the life of each project (starting with time 
elapsed = 40 days), you will have access to a series of plots 
providing information on the project. As the project 
progresses over time, the plots will be increasingly more 
meaningful in making your staffing decision. This and the 
next page explain how to interpret and use the plots in making 
your decisions. 



Plot on Project Size 

(refer to Chapter III, Figure 4) 

The figure below provides information on whether any schedule 
or budget slippage is a result of: (1) unexpected increases in 
the project's size (e.g., due to changes in users' 
requirements), or (2) that we initially underestimated the 
effort required to complete the project (e.g. because we 
underestimated its complexity). In the first case, the 
Perceived Total Project Size (PJBSZ) will increase over time. 
In the second case, the Perceived Total Project Cost (JBSZMD) 
will increase over time. 
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Plot on Project Staffing vs Schedule 
(refer to Chapter III, Figure 5) 

The figure below provides feedback on the trade-off between 
minimizing schedule over-runs versus minimizing cost 
over-runs. When a project runs into trouble, a manager can 
choose to stick with the project's schedule (SCHCDT) by 
increasing the workforce level ( FTEQWF ) . This always 
increases the cost of the project. On the other hand, a 
manager might desire to minimize his/her cost over-run, by 
avoiding an increase in the workforce level, and instead, 
increase the project's scheduled completion date. The 
indicated workforce level (WFINDC) is an estimate of the 
workforce needed to stay on schedule. 



Plot on Workforce Mix 

(refer to Chapter III, Figure 6) 

A good feedback on why your costs may be higher than expected 
is to evaluate your staffing decision. In general, staffing 
decisions have the greatest impact on productivity. First, a 
larger workforce (FTEQWF) will, in most cases, be less 
productive because of the increases in communication and 
training overheads. Second, the workforce mix (i.e., percent 
of experienced vs new staff in the workforce) will also have 
an impact. The larger the percentage of experienced people in 
the workforce (FRWFEX), the more productive the workforce. 



Plot on Workforce Productivity 
(refer to Chapter III, Figure 1) 

This figure provides information on the average productivity 
of the team. In general, your staffing decisions will affect 
productivity in two ways. The larger the workforce you 
assemble (FTEQWF), the lower the average productivity 
( ASSPRD ) . This is because in a larger workforce people 
"waste" more time communicating with team mates. This 
communication overhead is plotted above. The communication 
overhead (COMMOH) curve tells you the percentage of a person's 
time that is wasted (on average) in communication with others. 



★ ★ ★ 



You are now ready to Start PROJECT 1 



*** 



A2 Information provided to subject for first project - order 
of projects varied depending on subject' s group assignment 
( the order presented here is the same as the sequence 
given to subjects in Group 1: Ideal, Undersize, and 

then F ixeds ize/bad estimate) 

PROJECT 1 



Management's Initial Project Estimates 

Initial Estimate of Project Size: 
Initial Estimate of Project Cost: 
Initial Estimate of Project Duration: 
Maximum Tolerable Project Duration: 



1,067 Tasks 
3,721 Man Days 
413 Days 
479 Days 



A project is considered complete when there are less than 40 
days left for the project to be completed. That is, (New 
Estimate of Project Duration - Elapsed Time) < 40 days. 



Staffing Level Sought (in FTE) 



Please enter your staffing decisions, i.e.,(In Full Time 
Equivalent) below: 



Initial Estimate: 



Time elapsed 
Time elapsed 
Time elapsed 
Time elapsed 
Time elapsed 
Time elapsed 
Time elapsed 
Time elapsed 
Time elapsed 
Time elapsed 
Time elapsed 
Time elapsed 
Time elapsed 



- 40 days:_ 

- 80 days:_ 

- 120 days: 

- 160 days: 

- 200 days: 

- 240 days: 

- 280 days: 

- 320 days: 

- 360 days: 

- 400 days : 

- 440 days: 

- 480 days: 

- 520 days: 
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Time elapsed - 560 days: 
Time elapsed - 600 days: 
Time elapsed - 640 days: 



*** WHEN YOU ARE DONE, PLEASE CALL FOR A LAB ATTENDANT *** 

A3 Questions answered by all subjects after completion of 
first project 

1. Describe (in words, numbers, equation, etc) what decision 
rule you followed in deciding on the staffing level in 
this project: 



2. Please try to elaborate on the thinking process you went 
through in making your decisions in this project (use back 
of page if necessary) : 



3. How clear were the instructions regarding the task? 



4. To what extent was the report on the progress of the 
project helpful in improving your own decision? 



12 3 



4 5 6 7 8 9 



Not at all 
Clear 



Very 

Clear 



1 2 3 4 5 6 7 

Not at all 
Helpful 



8 9 



Very 

Helpful 
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A3 . 1 Additional question asked of subjects in feedforward 
condition only 

5. To what extent was the training provided before the 
experiment helpful in improving your own decision? 

123456789 
Not at all Very 

Helpful Helpful 



A3. 2 Additional question asked of subjects in cognitive 
feedback condition only 

6. To what extent was the graphical information provided on 
the progress of the project helpful in improving your own 
decision? 



12 3 4 

Not at all 
Helpful 


5 


6 


7 


8 


9 

Very 

Helpful 


*** PLEASE MOVE ON 


TO 


PROJECT 


2 


k k k 
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A4 Information provided to subject for second project - order 
of projects varied depending on subject' s group assignment 



Project 2 



Management's Initial Project Estimates 



Initial Estimate of Project Size: 
Initial Estimate of Project Cost: 
Initial Estimate of Project Duration: 
Maximum Tolerable Project Duration: 



397 Tasks 



380 Days 
441 Days 



1 , 460 Man Days 



A project is considered complete when there are less than 40 
days left for the project to be completed. That is, (New 
Estimate of Project Duration - Elapsed Time) < 40 days. 

Staffing Level Sought (in FTE) 

Please enter your staffing decisions, i.e.,(In Full Time 
Equivalent) below: 

Initial Estimate: 

Time elapsed - 40 days: 

Time elapsed - 80 days: 

Time elapsed - 120 days: 

Time elapsed - 160 days: 

Time elapsed - 200 days: 

Time elapsed - 240 days: 

Time elapsed - 280 days: 

Time elapsed - 320 days: 

Time elapsed - 360 days: 

Time elapsed - 400 days: 

Time elapsed - 440 days: 

Time elapsed - 480 days: 

Time elapsed - 520 days: 
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Time elapsed - 560 days: 

Time elapsed - 600 days: 

Time elapsed - 640 days: 

*** WHEN YOU ARE DONE, PLEASE CALL FOR A LAB ATTENDANT *** 

45 Questions answered by subject after completion of second 
project were the same as those answered after the first 

project 



*** PLEASE MOVE ON TO PROJECT 3 



* * * 



A6 Information provided to subject for third project - order 
of projects varied depending on subject' s group assignment 

Project 3 



Management's Initial Project Estimates 



Initial Estimate of Project Size: 
Initial Estimate of Project Cost: 
Initial Estimate of Project Duration: 
Maximum Tolerable Project Duration: 



1,866 Tasks 
2,972 Man Days 
362 Days 
420 Days 



A project is considered complete when there are less than 40 
days left for the project to be completed. That is, (New 
Estimate of Project Duration - Elapsed Time) < 40 days. 



Staffing Level Sought (in FTE) 

Please enter your staffing decisions, i.e.,(In Full Time 
Equivalent) below: 

Initial Estimate: 



Time 


elapsed 


- 


40 days: 


Time 


elapsed 


- 


80 days: 


Time 


elapsed 


- 


120 


days : 


Time 


elapsed 


- 


160 


days : 


Time 


elapsed 


- 


200 


days : 


Time 


elapsed 


- 


240 


days : 


Time 


elapsed 


- 


280 


days : 


Time 


elapsed 


- 


320 


days : 


Time 


elapsed 


- 


360 


days: 


Time 


elapsed 


- 


400 


days : 


Time 


elapsed 


- 


440 


days : 


Time 


elapsed 


- 


480 


days : 


Time 


elapsed 


- 


520 


days : 


Time 


elapsed 


- 


560 


days : 
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Time elapsed - 600 days: 

Time elapsed - 640 days: 

*** WHEN YOU ARE DONE, PLEASE CALL FOR A LAB ATTENDANT 



A7 Questions answered by subject after completion of third 
project were the same as those answered after the first 

project 



A8 Questions answered by cognitive feedback subjects after 
completion of entire experiment (subjects in the 
feedforward and outcome feedback answered only questions 
1, 7, 8, 9, 10, 11, 12, 13 and 14. 



1. In the projects that you just completed, did you 

(a) Use the project status report (Y/N)? 

(b) If you did, please describe how you used the 
information to make the staffing decision. 



2. In the projects that you just completed, did you 

(a) Use the project staffing report (Y/N)? 

(b) If you did, please describe how you used the 
information to make the staffing decision. 
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3. 



In the projects that you just completed, did you 

(a) Use the plot on project size (Y/N)? __ 

(b) If you did, please describe how you used the 
information to make the staffing decision. 



4. In the projects that you just completed, did you 

(a) Use the plot on staffing and schedule (Y/N)? 

(b) If you did, please describe how you used the 
information to make the staffing decision. 



5. In the projects that you just completed, did you 

(a) Use the plot on workforce mix (Y/N)? 

(b) If you did, please describe how you used the 
information to make the staffing decision. 
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6 . 



In the projects that you just completed, did you 

(a) Use the plot on workforce productivity (Y/N)? 

(b) If you did, please describe how you used the 
information to make the staffing decision. 



7. Have you in the past, participated in project management 
(Y/N)? 



8. If YES, to what extent was the task in this simulation 
similar to your previous experience? 

123456789 
Not at all Very 

Similar Similar 



9. 



How interesting was 
12 3 

Not at all 
Interesting 



the task you just performed? 

4 5 6 7 8 9 

Very 

Interesting 



10. How serious were you in performing the task? 



1 2 3 4 5 6 7 

Not at all 
Serious 



8 9 

Very 
Serious 



11. How clear were the instructions regarding the task, 
generally? 



1 2 3 4 5 6 7 

Not at all 
Clear 



8 9 

Very 
Clear 



12. How easy was the system to use? 

1 2 3 4 5 6 7 

Not at all 
Easy 



8 9 

Very 
Easy 
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13. Please give us some information about yourself (in 
absolute confidence. At no time will your name appear in the 
results. The data will only be used in an aggregate 

statistical sense). 

(a) Curriculum enrolled in: 

(b) Sex 

( c ) Age 

(d) Full time work experience 

(in years) 

(e) How long ago (in years) did 

you complete your 
undergraduate education? 

(f) How familiar are you with computers, generally? 

123456789 
Not at all Very 

Familiar Familiar 



(g) How many hours (per week) do you use computers? 



14. Your general comments regarding the simulation: 



*** END OF SIMULATION *** 
Thank you for your participation. 
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